Sampling Precision to Depth 9000: Evaluation Experiments at NTCIR-6

We describe evaluation experiments conducted by submitting retrieval runs for the Chinese, Japanese and Korean Single Language Information Retrieval subtasks of the Cross-Lingual Information Retrieval (CLIR) Task of the 6th NII Test Collection for IR Systems Workshop (NTCIR-6). We show that a Generalized Success@10 measure exposes a downside of the blind feedback technique that is overlooked by traditional ad hoc retrieval measures such as mean average precision, R-precision and Precision@10. Hence an important retrieval scenario, seeking just one item to answer a question, is not properly evaluated by the traditional ad hoc retrieval measures. Also, for each language, we submitted a one-percent subset of the first 9000 retrieved items to investigate the frequency of relevant items at deeper ranks than the official judging depth of 100. The results suggest that, on average, less than 60% of the relevant items for Chinese and less than 80% for Japanese are assessed.